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ABSTRACT 

The  inereasing  eomplexity  of  today’s  information  environment  is  refleeted  in  the  inereasing 
eomplexity  of  military  operations.  A  brief  look  at  the  evening  news  reveals  organizational  and 
teohnologieal  eomplexity  that  was  unheard-of  even  a  generation  ago:  Improved  explosive 
deviees  detonated  via  eel  phones;  unmanned  vehieles  both  ground  and  air;  handheld 
eommunieation  deviees  on  virtually  every  eombatant;  ubiquitous  and  near-instantaneous  media 
eoverage  of  even  the  smallest  military  aetions.  All  of  these  faetors,  and  more,  ean  eritieally 
impaet  the  sueeess  of  military  operations.  Understanding  this  new  environment  to  faeilitate 
mission  sueeess  is  truly  a  ehallenge. 

Just  as  with  operations  in  the  real-world  taetioal  environment,  our  ability  to  test  military 
equipment  depends  eritieally  on  our  ability  to  rapidly  understand  eauses  and  effeets  among 
numerous  variables  represented  by  extremely  high-volume  data  of  several  types  (sueh  as 
numerieal,  visual,  boolean,  and  textual).  Traditional  analysis  methods,  whieh  often  eenter  on 
very  effieient  mining  of  single  data  sources,  are  increasingly  inadequate  to  meet  this  challenge, 
and  advanced  means  of  visualization  and  understanding  are  required. 

While  our  initial  focus  was  on  analysis — understanding  data  produced  by  test  events — ^we  have 
quickly  come  to  the  realization  that  operational  test  and  evaluation  will  use  visualization 
technologies  not  only  for  analysis,  but  for  planning  and  control  of  test  events.  This  control  is  not 
only  of  live  operations,  but  increasingly  of  sophisticated  federations  of  models  and  simulations 
that  support  the  conduct  of  our  tests.  The  ability  to  develop,  integrate,  and  operate  these 
federations  directly  reflects  the  complexity  of  actual  military  operations,  and  depends  on  our 
ability  to  understand  the  relationships  among  multiple  software  applications,  each  of  which  is 
extremely  complex  in  its  own  right.  Hence,  technical  control — such  as  of  the  simulation 
federation,  of  test  and  evaluation  networks,  or  of  data  collection  applications — becomes 
increasingly  challenging,  yet  potentially  tractable  due  to  the  use  of  visualization  technologies 
similar  to  those  in  use  in  the  commercial  world. 
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Introduction 


The  increasing  complexity  of  today’s  information  environment  is  reflected  in  the  increasing 
complexity  of  military  operations.  A  brief  look  at  the  evening  news  reveals  organizational  and 
technological  complexity  that  was  unheard-of  even  a  generation  ago:  Improved  explosive 
devices  detonated  via  cel  phones;  unmanned  vehicles  both  ground  and  air;  handheld 
communication  devices  on  virtually  every  combatant;  ubiquitous  and  near-instantaneous  media 
coverage  of  even  the  smallest  military  actions.  All  of  these  factors,  and  more,  can  critically 
impact  the  success  of  military  operations.  Understanding  this  new  environment  to  facilitate 
mission  success  is  truly  a  challenge. 

Just  as  with  operations  in  the  real-world  tactical  environment,  our  ability  to  test  military 
equipment  depend  critically  on  our  ability  to  rapidly  understand  causes  and  effects  among 
numerous  variables  represented  by  extremely  high-volume  data  of  several  types  (such  as 
numerical,  visual.  Boolean,  and  textual).  Traditional  analysis  methods,  which  often  center  on 
very  efficient  mining  of  single  data  sources,  are  increasingly  inadequate  to  meet  this  challenge, 
and  advanced  means  of  visualization  and  understanding  are  required. 

It  is  axiomatic  in  software  engineering  that  an  understanding  of  the  problem  domain  is  an 
essential  first  step  in  crafting  a  solution*,  and  this  problem  is  no  different.  This  paper  describes 
the  problem  domain  as  seen  by  the  operational  tester,  and  our  initial  steps  towards  solutions.  We 
first  describe  the  problem  domain  (section  1),  followed  by  an  overview  (section  2)  of  our  process 
towards  finding  solutions;  a  brief  description  of  our  initial  operational  architecture  (section  3); 
and  (section  IV)  a  presentation  of  a  few  of  the  promising  technology  solutions  that  we  have 
found,  using  them  to  illustrate  features  of  benefit  to  the  operational  tester. 

I.  The  Problem  Domain 

While  our  initial  focus  was  on  analysis — understanding  the  data  produced  by  test  events — ^we 
have  quickly  come  to  the  realization  that  operational  test  and  evaluation  can  use  visualization 
technologies  for  other  purposes.  Just  as  in  the  operational  force  or  in  the  commercial  world  , 
visualization  technologies  can  be  used  to  improve  our  ability  to  control  a  test  event — both  in 
planning  an  event,  and  in  monitoring  and  understanding  its  execution.  Further,  these 
technologies  can  assist  in  both  operational  control  (that  is,  controlling  the  execution  of  the  event 
by  test  team  and  test  unit)  and  technical  control  (controlling  the  hosts  and  applications  that 
support  the  event,  such  as  by  collecting  data). 

Our  work  to  date  indicates  that  the  challenge  of  operational  and  technical  control  is  in  many 
ways  quite  similar  to  that  faced  by  operational  units:  The  operational  tester  has  the  same 
challenges  of — 

•  creating,  analyzing,  and  choosing  among  courses  of  action,  while— 

o  allocating  forces  to  terrain  and  to  missions, 

o  planning  networks  while  considering  the  mission,  the  terrain,  and  the  weather, 

•  rehearsing,  and 

•  monitoring  execution  such  that  needed  changes  can  be  made  in  a  timely  manner. 


We  examine  each  of  these  in  turn. 


In  terms  of  operational  control,  the  tester’s  task  of  creating,  analyzing,  and  choosing  among 
courses  of  action  is  very  similar  to  the  operational  warfighter’s.  Two  key  similarities  are  the 
tasks  of  allocating  forces  to  terrain  and  to  missions  and  planning  networks  while  considering  the 
mission,  the  terrain,  and  the  weather.  The  tester,  however,  must  face  the  added  complexity  of 
performing  this  task  both  for  the  test  unit  (to  determine  the  probable  tactical  execution  of  the  test, 
which  is  needed  to  plan  data  collection  and  other  test  operations)  and  for  the  test  team  (to 
determine  the  best  means  to  collect  data  while  minimizing  interference  with  the  tactical 
scenario).  An  increasingly  important  point  for  the  tester,  just  as  for  the  warfighter,  is  the 
placement  of  sensors  (in  test  terms,  instrumentation)  such  that  they  will  collect  the  desired  data  at 
the  proper  time,  and  are  not  impeded  by  terrain  or  distance.  Figure  1,  below,  illustrates  this 
challenge  in  simple  terms:  The  tester  cannot  afford  to  place  data  collection  sensors  at  every 
point  on  the  test  — bttlefield,”  so  must  choose  carefully  where  the  action  will  take  place,  and 
where  he  places  his  sensors.  In  this  scenario,  the  friendly  unit  may  choose  axes  of  advance 
represented  by  both  the  solid  and  the  dashed  arrows — or  may  choose  to  send  forces  along  both. 
Modeling  of  terrain  and  electromagnetic  energy  propagation,  coupled  with  simulation  of  the 
ground  combat  operations  of  the  units  in  the  sketch,  will  allow  the  tester  the  best  opportunity  to 
control  both  the  event  and  his  sensors  to  get  the  most  good  data  from  the  event. 


The  tester’s  challenge  of  technical  control  is  again  very  similar  to  that  of  users  in  other  domains. 
Planning  networks  while  considering  the  mission,  the  terrain,  and  the  weather  is  also  a  task  very 
similar  to  that  faced  by  operational  warfighters.  The  increasingly  networked  nature  of  test 
instrumentation  and  of  simulation  federations^  creates  a  potential  requirement  to  establish  and 
maintain  network  connectivity  among  all  -test  players”  such  that  engagements  can  be  simulated 
and  weapons  effects  assessed  rapidly.  Since  many  of  the  test  players  may  be  in  combat  vehicles. 


this  places  a  severe  ehallenge  at  the  feet  of  network  planners.  Similarly,  networks  to  support  test 
team  operations — sueh  as  data  collection,  or  data  analysis — must  take  into  aecount  not  only 
terrain  and  weather  but  the  probable  taetical  execution  of  the  test.  Those  readers  familiar  with 
mounted  eombat  operations  are  probably  familiar  with  the  effeets  of  a  heavy  vehicle  running 
over  an  exposed  telephone  or  network  cable. 

The  test  team  will  also  in  many  oases  faoe  the  challenge  of  comprehending  and  controlling 
applications  exchanging  extremely  high  volumes  of  critical  data.  For  instance,  many  tests  now 
use  an  integrated  live  and  oonstructive  simulation  environment,  where  information  is  exohanged 
via  protocols  like  the  High  Level  Arohitecture  (HLA)  or  the  Test  and  Training  Enabling 
Architecture  (TENA):  disconnects”  in  data  format  or  content  among  these  applioations  can 
cause  exeoution  to  fail,  or — almost  as  oritioal — present  inoorrect  information  to  the  test  unit  or 
the  test  team. 

In  all  these  eases,  the  ability  to  — vinally  rehearse”  the  test  operation — ^whether  in  operational  or 
technical  terms,  or  both — will  be  invaluable  to  the  tester,  in  analyzing  eourses  of  action  (did  I 
place  the  instrumentation  correctly?”),  in  ensuring  test  team  understanding  of  the  mission  and  its 
exeoution,  and  in  ensuring  the  oorreotness  of  execution. 

Monitoring  test  operations  oan  again  be  understood  in  light  of  its  similarity  to  the  task  of 
monitoring  military  operations.  Again,  the  similarity  is  tempered  by  the  additional  complexity  of 
monitoring  both  test  unit  and  test  team  operations;  and  the  latter  is  in  terms  of  both  operational 
and  technical  control.  Increasingly"^,  the  tester  is  challenged  to  understand  the  status  of  his 
instrumentation  and  data  colleetion  in  near  real  time,  so  that  progress  ean  be  measured  and 
execution  changed — or  suspended — in  response  to  the  speeifie  conditions  encountered. 

Analysis  of  the  data  gathered  is  perhaps  the  most  challenging  of  the  visualization  tasks:  It  oan  be 
thought  of  as  consisting  of — 

•  Authenticating  data, 

•  Ordering  data, 

•  Conduoting  extended  analysis  of  single-factor  data, 

•  Conducting  extended  analysis  of  multi-factor  data,  and 

•  Applying  Evaluative  Military  Judgment^ 

Authenticating  data  is  the  task  of  confirming  the  test  data,  correoting  it  as  necessary  (such  as  by 
identifying  and  discarding  invalid  or  redundant  data  points),  and  recording  the  results  of  the 
prooedure  in  usable,  agreed-to  formats.^  In  operational  tests,  this  task  is  the  oharge  of  the  Data 
Authentioation  Group  or  DAG.  The  DAG  is  often  oonfronted  with  enormous  volumes  of  data^, 
and  usually  has  limited  time  to  perform  its  tasks.  Eurther,  DAGs  may  review  data  of  very 
different  types  (video,  military  messages,  network  traffie)  but  relating  to  the  same  test  event. 
These  requirements  immediately  result  in  the  requirement  for  new  visualization  techniques  to 
comprehend  extremely  high-volume  data,  and  the  need  to  comprehend  this  data  from  multiple 
sourees.  These  ehallenges  are  discussed  in  more  detail  below. 


Ordering  data  is  the  task  of  arranging  data  in  convenient  order  for  further  proeessing.  While  this 
may  seem  simple  in  coneept — for  instance,  ordering  military  messages  by  the  time  of  their 


arrival  at  a  given  node — in  praetiee  it  ean  be  extremely  eomplieated.  Whieh  of  thousands  of 
nodes,  for  instanee,  are  relevant  for  arrival  times?  Given  sophistieated  routing  topologies  often 
found  in  modern  military  networks,  are  routing  protocols  and  their  implementation  important  to 
those  arrival  times?  What  were  the  relations  of  time-  and  location-dependent  battlefield  factors 
(such  as  electronic  attack  or  jamming”)  with  message  arrival  times?  These  are  only  a  few  of  the 
many  conceivable  challenges  to  comprehending  data  and  -arranging. .  .for  further  processing.” 

Extended  analysis  goes  beyond  primary  statistical  tests  to  (for  instance)  combine  analytic  results 
from  different  sources,  to  apply  queuing  or  inventory  theory  or  decision  analysis  techniques,  or 

o 

to  use  mathematical  models  or  combat  simulations.  The  analyst  may  work  with  numerous  high- 
volume  data  sources  in  different  combinations  using  different  tests  or  algorithms.  The  potential 
combinations  of  data  could  be  virtually  limitless:  In  a  large  combined- arms  test  such  as  of  the 
Future  Combat  System  or  ECS,  an  analyst  might  have  multiple  video  sources  (from  cameras 
placed  on  the  test  battlefield,  or  of  individual  system  user  interfaces),  numerous  audio  sources 
(from  test  unit  radio  nets),  several  streams  of  military  message  traffic  (such  as  for  situational 
awareness,  or  for  command  and  control,  from  any  one  of  many  units),  application  logs,  and 
signal-strength  data  from  the  electronic  spectrum  collected  at  several  or  many  points  on  the 
battlefield.  Many  factors  represented  by  these  data  sources  could — singly  or  in  combination — 
conceivably  contribute  to  a  unit’s  effectiveness  in  the  test,  resulting  in  questions  such  as  the 
following: 

•  What  was  the  effect  of  electronic  attack  on  the  quality  of  information  presented  to  a 
certain  user? 

•  What  was  the  effect  of  location  on  information  quality? 

•  Did  the  quality  of  information  available  at  a  certain  time  affect  a  commander’s  decision? 

These  questions,  and  many  more,  will  tax  the  analyst  to  comprehend  very  large  data  sets,  often 
of  disparate  types,  and — most  crucially — comprehend  their  combined  effect  on  the  results  of  the 
test. 

Finally,  applying  evaluative  military  judgment  is  perhaps  the  most  complicated  analysis  task  of 
all.  The  analyst  must  not  only  reach  conclusions  about  the  overall  effectiveness  and  suitability  of 
a  given  system  from  data  that  is  often  brutally  complicated,  he  or  she  must  be  able  to 
comprehend  or  even  challenge  the  work  of  the  analysts  in  all  previous  steps:  Have  all  relevant 
factors  been  considered?  What  is  the  sensitivity  of  the  conclusion  to  each  factor  in  the  test? 

What  challenges  have  been  raised  to  the  analysis,  and  have  they  been  fully  addressed?  This  step 
emphasizes  the  requirement — already  suggested  by  the  earlier  work — to  rapidly  and  easily 
compose  and  comprehend  queries  against  very  large  data  sets,  and  in  similar  fashion  to  integrate 
the  queries  and  their  results. 


II.  The  Solution  Approach 

OTC’s  Transformation  Technology  Directorate  (TTD)  chose  to  approach  the  data  visualization 
problem  through  an  integrated  process  team  (IPT).  The  IPT  in  turn  chose  to  approach  this 
complex  problem  in  multiple  phases  that  will  take  place  over  FY09  and  FYIO,  thereby  allowing 
Visualization  IPT  members  to  analyze  multiple  problem  sets  (instantiated  as  the  requirements  of 


specific  tests)  while  examining  numerous  potential  solutions.  The  IPX  chose  to  use  as  its  lingua 
franca  the  Department  of  Defense  Architecture  Framework  (DoDAF),  Version  1.5,  to  document 
the  processes  and  systems  associated  with  visualization  issues. 

The  IPX  quickly  identified  the  visualization  needs  of  operational  control,  technical  control,  and 
data  analysis  described  above;  what  was  needed,  however,  was  a  more  detailed  view  of  the 
problem  domain — down  to  the  low-level  tasks,  and  further  down  to  the  visualization 
requirements  associated  with  those  tasks.  The  DoDAF  Operational  View  -  5  (OV-5,  Operational 
Activity  Mode),  was  used  to  begin  this  process;  the  team  chose  to  depict  the  OV-5  as  an  Activity 
Hierarchy^  to  facilitate  organizing  it  in  a  logical,  easy-to-understand  format.  This  is  described 
further  in  the  next  section.  We  believe  our  current  OV-5  is  a  significant  advance,  but  obviously 
represents  only  a  first  step  in  defining  the  tester’s  requirements  for  visualization. 

After  the  initial  requirements  determination,  the  IPX  will  verify  the  requirements  through  the 
target  audience — OTC’s  test  teams.  This  process  will  require  the  demonstration  of  potential 
solutions  in  order  to  elicit  the  thoughts  of  the  user  on  the  emerging  capabilities  and  the  feedback 
on  how  demonstrated  capabilities  may  tie  back  to  known  requirements,  enhance  test  capabilities, 
or  create  unnecessary  overhead. 

The  capabilities  of  the  potential  solutions  will  be  recorded  in  a  DoDAF  Systems  View  -  4  (SV-4) 
to  allow  the  IPX  to  compare  tool  capabilities  against  known  requirements.  After  the  requirements 
are  verified  and  the  correlating  solutions  are  identified,  a  knowledge  repository  containing  the 
requirements  and  the  corresponding  possible  solutions  that  will  be  accessible  throughout  OTC 
for  testers,  analysts  and  technologists  will  be  created.  The  last  phase  of  the  Visualization  IPX 
effort  will  be  to  provide  an  overall  demonstration  of  visualization  capabilities  for  key 
requirements,  facilitating  selection  of  the  tools  that  best  meet  those  requirements  when  the 
criteria  of  affordability,  tailorability,  extensibility,  and  usability  are  taken  in  consideration. 

III.  The  Operational  Architecture 

When  the  IPX  began  assessing  the  requirements  for  visualization  across  a  broader  spectrum  than 
data  analysis,  the  user’s  -task  list”  of  course  became  much  larger.  It  appears  to  us  that 
visualization  capabilities  may  support  tasks  such  as  configuring  complex  system-of-systems 
environments,  and  for  controlling  test  operations.  Tasks  such  as  these  defined  the  components 
and  sub-components  of  the  operational  architecture  for  the  IPX  effort.  The  team  chose  to  group 
the  sub-components  in  a  manner  roughly  corresponding  to  major  activities  in  a  test  event,  as 
shown  in  Figure  2  below.  This  served  as  the  first  level  of  the  activity  hierarchy. 


Figure  2.  First  Level  Aetivity  Hierarehy. 

The  subcomponents  of  the  Operational  Control  task  are  shown  below  in  Figure  3.  Turning  first 
to  the  Operation  Planning  subcomponent,  we  further  divide  the  task  into  Scenario  Generation 
and  Comprehensive  Event  Rehearsal.  While  various  simulations  provide  graphical  means  to 
build  their  scenarios — as  well  they  should  given  the  size  and  complexity  of  even  single¬ 
simulation  scenarios — ^we  are  unaware  of  any  current  capability  to  do  this  for  the  much  more 
complicated  case  of  a  scenario  for  an  integrated  LVC  test  environment.  Further,  we  believe  the 
ability  to  — vinally  rehearse”  this  scenario  will  greatly  enhance  the  test  officer’s  understanding 
of  the  event,  and  his  identification  of  risks  to  the  test  as  was  briefly  mentioned  in  Section  1 . 

In  the  Operation  Execution  phase,  an  integrated  common  operational  picture  (Integrated  COP) 
allows  the  test  officer  to  understand  the  state  of  his  -test  battle  space,”  including  both  live  and 
virtual  or  constructive  entities.  The  ability  to  see  Entity  Attributes  —  exactly  what  that  entity  is, 
whether  it  is  live  or  constructive,  and  its  state  -  facilitates  understanding  of  the  state  of  test 
execution  and  thereby  supports  decision-making.  An  immersive  3D  Battle  space  for  virtual 
entities  can  simulate  actual  tactical  operations  center  (TOC)  assets  (such  as  the  video  from  an 
unmanned  aerial  system  (UAS)),  or  stimulate  a  role  player  to  provide  quick  and  more  accurate 
reactions  to  events  in  the  scenario. 

It  is  common  practice  to  conduct  After  Action  Reviews  (AARs)  in  the  Post  Operation  -Wrap-up” 
for  both  the  test  team  and  the  test  unit.  While  the  focus  of  each  will  be  on  the  execution  of  the 
event,  the  test  team  will  of  course  focus  on  test  team  activities  (such  as  data  collection,  or  the 
execution  of  the  simulation  environment)  while  the  test  unit  will  focus  on  its  own  execution  of 
military  tasks.  In  both  cases,  the  ability  to  playback  events  in  the  scenario  adds  greater  fidelity  to 
the  AAR,  facilitating  understanding  by  the  unit  and  the  test  team  of  what  happened,  its  effect  on 
the  operation,  and  why  it  happened. 


Figure  3.  Operational  Control  Aetivity  Hierarehy. 


The  Teehnieal  Control  Aetivity  Hierarehy  is  shown  below  in  Figure  4.  The  IPT  ehose  to 
structure  it  based  on  the  tasks  we  have  performed  in  configuring  a  live,  virtual,  and  constructive 
(LVC)  test  support  architecture.  The  key  tasks  we  have  identified  include  validating  the 
interfaces  among  LVC  components;  network  monitoring;  and  monitoring  system  health.  While 
the  last  two  tasks  may  appear  mundane — at  least  to  those  who  operate  and  manage  systems  in  a 
production  environment — the  first  is  somewhat  peculiar  to  system-of-systems  integration  such  as 
is  done  in  a  simulation  federation.  In  particular,  we  encounter  the  need  to  validate  the 
correctness  of  data  exchanged  among  the  systems — both  in  terms  of  its  syntax  or  format,  and  in 
terms  of  its  semantics  or  underlying  meaning.  The  ability  to  visually  comprehend  and  analyze 
these  exchanges  could  reduce  configuration  time,  greatly  expedite  the  Verify  and  Validation 
effort  of  the  architectures,  and  in  turn  allow  rapid  response  to  any  interoperability  issues  that  may 
arise  during  execution. 


Figure  4.  Technical  Control  Activity  Hierarchy. 


While  perhaps  mundane  to  many  readers,  the  Network  Monitoring  and  System  Performance 
components  are  just  as  important  to  the  control  of  the  test  support  architecture.  Monitoring 
network  resources  such  bandwidth,  and  identifying  the  protocols  used,  are  much  more  easily 
done  graphically — such  as  is  currently  done  in  the  commercial  sector — when  the  network  is 
complex  and  traffic  volume  is  high.  (We  suspect  that  this  subcomponent  might  be  more  finely 
subdivided  in  later  versions  of  the  architecture,  but  believe  this  initial  division  helps  shed  light  on 
the  tasks  that  might  be  performed.)  These  capabilities  allow  test  teams  to  understand  the  status 
of  its  network  in  time  to  make  corrections  as  necessary — ensuring  that  data  reaches  the 
processing  center  and  simulation  information  reaches  the  unit  under  test.  System  performance 
metrics,  such  as  CPU  or  Memory  Utilization,  or  CPU  processing  queues,  form  key  indicators  of 
the  health  of  critical  nodes,  allowing  the  team  to  shift  processing  loads  or  change  collection 
schemes  as  required.  Data  Collection  Node  Status  is  the  indication  of  the  status  (functioning, 
powered  off,  in  or  out  of  communication)  of  the  various  data  collection  devices  in  the  test.  Just 
as  in  the  case  of  network  monitoring,  this  is  simple  to  do  for  a  few  systems,  but  very  difficult  for 
hundreds  or  thousands — and  again,  as  is  done  in  the  commercial  sector,  graphical  displays 
facilitate  rapid  understanding  of  the  situation  and  help  decision-making. 


The  process  of  data  analysis,  already  briefly  described  in  Section  1,  is  shown  in  Figure  5,  below. 
The  two  subcomponents.  Preliminary  Data  Analysis  and  Post  Event  Data  Analysis,  roughly 
correspond  to  the  tasks  of  the  test  and  the  evaluation  team  in  Army  operational  testing;  but  more 
specifically  correspond  to  the  immediate  steps  taken  to  ensure  data  quality  (Preliminary)  and  to 
analyze  the  data  in  depth  (Post  Event).  The  Quick  Look  step  is  taken  to  ensure  the  presence  of 
readable  data  on  the  media  delivered  from  the  test  site,  and  depending  on  time  available  may 


include  more  in-depth  quality  checks  of  the  data,  such  as  looking  for  and  analyzing  anomalous 
data  values. 

Visualization  technologies  show  their  great  value  in  the  next  two  steps,  Authenticating  and 
Ordering  Data.  As  previously  mentioned,  the  Data  Authentication  Group  or  DAG  is  often 
confronted  with  enormous  volumes  of  data,  and  has  little  time  to  perform  its  work;  this  challenge 
is  exacerbated  by  the  presence  of  multi-source  data  such  as  from  communications  logs,  video 
cameras,  radar  traces,  and  textual  data.  It  is  here  that  integrated-query  technologies,  such  as  the 
Sensor  Data  and  Analysis  Framework  described  below,  are  particularly  useful:  The  DAG  can 
rapidly  and  easily  compare  data  from  several  sources,  bounded  by  time,  position,  or  both;  and 
can  therefore  rapidly  identify  and  diagnose  apparently  anomalous  values.  Ordering  data  also 
shows  strong  potential  for  visualization  technologies:  When  the  data  is  purely  and  obviously 
only  time-ordered,  little  assistance  is  necessary,  but  in  today’s  sophisticated  networks  this  simple 
case  is  extremely  rare.  Visualization  of  a  network  topology  and  its  traffic,  for  instance,  supports 
good  decision-making  in  support  of  ordering,  yielding  information  on  (for  instance)  which  nodes 
had  the  highest  traffic  volumes,  which  interfaces  were  the  busiest,  and  so  forth.  Again,  just  as  is 
commonly  done  in  the  commercial  sector  this  can  also  assist  in  troubleshooting  slow  or 
otherwise  problematic  information  exchanges — help  which  can  be  of  great  value  to  the  system’s 
project  manager  or  development  and  engineering  teams. 


Figure  5.  Data  Analysis  Activity  Hierarchy. 


Post  Event  Data  Analysis  takes  an  in-depth  look  at  the  test  data  to  produce  overall  results  and  to 
arrive  at  conclusions,  such  as  on  the  effectiveness  or  suitability  of  a  given  weapons  system. 
Broadly  speaking,  it  can  be  decomposed  into  the  basic  tasks  of  single-  and  multi-factor 
analysis'*^,  plus  the  heterogeneous  task  of  applying  evaluative  military  judgment  to  the  analysis 
results.  In  all  these  cases,  visualization  can  assist  what  would  otherwise  be  a  daunting  task. 

Single-factor  analysis,  as  its  name  implies,  focuses  on  the  effects  of  a  single  causative  battlefield 
factor,  such  as  the  presence  or  absence  or  electronic  attack  (^mming”),  or  the  strength  of  the 
jamming  signal.  — Sigle  factor,”  however,  does  not  mean  -simple”:  This  analysis  may  need  to 
comprehend  the  effect  of  that  single  factor  over  thousands  of  battlefield  entities,  each  of  which 
may  encounter  numerous  environmental  or  potentially  confounding  factors.  Here  again, 
advanced  visualization  technologies  can  help  the  analyst  by  displaying,  for  instance,  color  codes 
for  ranges  of  attribute  values,  allowing  the  analyst  to  quickly  discern  patterns  in  the  data  which 
may  be  related  to  time,  location,  or  distance  from  a  given  emitter. 

The  presence  of  other  factors  of  course  leads  to  multi-factor  analysis,  which  extends  the  task  to 
examine  the  combined  effect  of  multiple  factors,  such  as  weather  plus  electronic  attack,  or 
weather  plus  electronic  attack  plus  emitter  location.  Given  the  great  proliferation  of  potential 
data  sources,  and  the  amount  of  data  they  may  generate,  the  analyst  is  clearly  challenged  to  come 
to  terms  with  the  data  he  is  examining,  to  say  nothing  of  arriving  at  and  defending  his 
conclusions.  Visualization  solutions  for  this  process  must  allow  rapid  and  tailorable  queries, 
facilitating  many  different  explorations  of  the  data  and  many  tests  of  the  analyst’s  conclusions. 

Finally,  applying  evaluative  military  judgment  may  apply  the  techniques  of  all  the  preceding 
analysis  steps  as  the  evaluator  checks  his  analysts’  tests  and  conclusions,  and  asks  for  or 
performs  sensitivity  analyses.  This  will  place  a  premium  on  the  ability  of  visualization  solutions 
to  store  queries  and  query  results,  both  within  single  sources  and  across  multiple  sources. 

IV.  Promising  Potential  Solutions 

Because  we  are  at  a  very  early  stage  in  our  journey,  it  is  clear  to  us  that  both  our  requirements 
and  our  potential  solutions  will  change  significantly  over  time.  Nevertheless,  the  similarities 
described  above  between  testing  and  other  problem  domains  have  greatly  facilitated  our  search 
for  promising  potential  solutions  to  these  challenges.  A  few  of  these,  chosen  for  their  feature 
sets  which  illustrate  key  parts  of  the  problem  and  solution  domains,  are  described  below. 

The  Sensor  Data  and  Analysis  Framework  (SDAF),  developed  by  a  team  at  the  MITRE 
Corporation,  offers  the  unique  feature  of  performing  integrated  queries  on  both  streaming  (such 
as  sensor  data)  and  persistent  (such  as  RDBMS)  data  sources.  Based  on  a  service  oriented 
architecture  (SOA),  SDAF  allows  the  flexible  combination  of  data  sources,  data  operators  (such 
as  stream  mining  algorithms),  and  various  visualization  clients.  A  sketch  of  the  SDAF 
architecture  is  shown  below  at  Figure  6. 


SOAF  SERVICE -ORIENTED  ARCHITECTURE  (SOA)  OVERVIEW  v2  0 
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Figure  6.  SDAF  Architecture. 

Notably,  SDAF  was  not  developed  in  response  to  the  needs  of  testers,  but  rather  in  response  to 
the  needs  of  operational  intelligence  analysts  confronted  with  extremely  high-volume  data  from 
multiple  sources  such  as  ground  moving-target  indication  (GMTI),  video,  textual  reports,  and 
photographs.  The  Air  Force  is  currently  using  SDAF  for  data  characterization;  OTC  is 
studying  its  use  in  both  test  control  and  in  data  analysis  functions. 

A  sample  SDAF  display,  integrating  position-location  data  with  real-time  casualty  assessment 
(RTCA)  data  is  shown  at  Figure  7.  In  this  implementation,  the  analyst  can  choose  any  vehicle 
location  from  a  -snail  trail”  of  its  locations  over  time  and  be  immediately  shown  all  position- 
location  attributes  for  it  (northing  and  easting  grid  locations,  altitude,  and  exact  time);  he  or  she 
can  then  query  the  RTCA  database  to  find,  for  instance,  all  shots  taken  by  or  at  this  vehicle  in  a 
tailorable  time  period,  including  shooter,  target,  weapon  and  ammunition,  and  engagement  result. 


Figure  7.  Example  SDAF  Display. 

While  the  use  of  SDAF  in  data  analysis  would  seem  fairly  obvious,  with  many  possible  uses 
similar  to  the  example  above,  its  potential  use  as  a  test  control  tool  requires  elaboration. 
Interviews  with  test  officers"  indicate  that — in  addition  to  the  basic  requirement  to  monitor 
vehicle  or  personnel  locations  on  a  map  display — the  operational  test  officer  is  challenged  to 
monitor  and  comprehend  the  detailed  status  of  his  data  collection,  and  to  compare  it  against  that 
which  the  test  and  evaluation  team  expects  for  the  event;  Are  all  collection  devices  working? 
Are  they  collecting  the  desired  kinds  of  data,  and  are  they  collecting  enough  data  to  meet  sample 
size  requirements?  Given  the  constantly-increasing  costs  of  military  operations,  the  answers  to 
these  last  two  questions  may  mean  the  difference  between  success  and  failure  for  an  extremely 
expensive  trial — or  may  offer  the  opportunity  to  terminate  the  trial  early  and  -give  back” 
valuable  time  to  the  test  unit  commander. 

Another  tool  of  note,  developed  by  Texas  A&M  University  under  the  Army’s  University  XXI 
research  program,  emphasizes  the  development  of  visual  analytics: 

. .  .the  science  of  analytical  reasoning  facilitated  by  interactive  visual  interfaces. 

People  use  visual  analytics  tools  and  techniques  to  synthesize  information  and 
derive  insight  from  massive,  dynamic,  ambiguous,  and  often  conflicting  data; 
detect  the  expected  and  discover  the  unexpected;  provide  timely,  defensible,  and 
understandable  assessments;  and  communicate  assessment  effectively  for  action." 

The  University  XXI  visualization  tool  is  based  on  an  interactive  software  framework  capable  of 
interfacing  with  a  variety  of  data  sources,  such  as  relational  databases  or  text  files.  A  sketch  of 
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the  tool’s  architecture  is  shown  in  Figure  .  As  Figure  8  implies,  the  tool  provides  an  extensible 
set  of  data  operators  and  processing  modules,  which  produce  cached  intermediate  data  that  can 
then  be  interactively  explored  using  various  visualization  techniques. 
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Figure  8.  University  XXI  Data  Visualization  Architecture. 


Of  particular  note  in  this  effort  is  the  emphasis  on  a  large  and  growing  set  of  linked  visual 
representations  of  the  data  set:  As  of  October  2008,  the  set  included  various  bar,  line,  radial  axis, 

1  Q 

parallel  axis,  frequency,  availability,  event  timeline,  and  geospatial  displays.  University  XXI 
researchers  are  using  these  and  other  techniques  to  explore  analysts’  comprehension  of  very 
large  data  sets,  such  as  the  terabyte-sized  database  from  a  recent  communications  system  test,  or 
the  results  of  ground-combat  tests  involving  large  numbers  of  direct- fire  engagements.  A  sample 
multi -view  visualization  of  a  series  of  information  exchanges  is  shown  at  Figure  9. 


Figure  9.  Multi-View  Visualization  of  Communieations. 


Of  partieular  note  in  eaeh  of  these  potential  solutions  is  not  only  the  rieh  visual  display  of 
information,  but  the  thoughtful  arrangement  of  features  in  the  user  interfaee  so  that  the  analyst 
ean  quickly  -drill  down”  on  an  item  of  interest  to  reveal  the  data  underneath.'"^  The  SDAF  team, 
for  instance,  built  a  visualization  of  a  ground-combat  test  integrating  position-location  data, 
RTCA  data,  reliability,  availability,  and  maintainability  (RAM)  data,  and  textual  observations; 
all  data  was  accessible  in  seconds  via  a  few  computer  mouse  clicks  that  tailored  queries  by  data 
type,  by  location,  and  by  time  window.  By  contrast,  analysts  are  rightfully  dissatisfied  with 
solutions  that  provide  only  a  bare  visual  representation  of,  say,  entity  locations  over  time  with  no 
access  to  the  huge  volume  of  relevant  data  associated  with  them.'^ 

V.  Conclusion. 

We  have  seen  how  the  great  complexity  of  today’s  military  operations  is  reflected  in  the 
increasing  complexity  of  the  operational  testing  environment,  where  the  tester  is  confronted  with 
extremely  high-volume  data  of  numerous  and  disparate  types.  Visualization  technologies  and 
techniques  are  not  merely  interesting  -  they  are  crucial  for  the  rapid  comprehension  of  this  new 
environment  of  extreme  complexity.  While  traditional  analysis  methods  appear  to  have  reached 
the  limits  of  their  effectiveness  in  this  new  regime,  technologies  developed  both  within  and 
outside  of  the  operational  test  domain  offer  great  promise,  and  illustrate  feature  sets  that  inform 
the  issue.  We  believe  our  beginnings  of  an  operational  architecture,  and  the  initial  requirements 
we  are  developing  from  it,  are  necessary  initial  steps  to  choosing  the  right  solutions — and  we 
invite  comments  and  input  from  other  practitioners!  The  conclusion  is  just  the  beginning  of  the 


Visualization  IPX  effort  and  while  our  IPX  has  taken  some  important  initial  steps,  they  are  just 
that — initial  steps.  We  have  a  long  yet  fase mating  journey  ahead. 


*  Dean  Leffingwell  and  Don  Widrig,  Managing  Software  Requirements:  A  Unified  Approach,  Boston:  Addison- 
Wesley,  2000,  p.  32. 

^  Many  readers  will  easily  recognize  the  visualization  techniques  and  technologies  in  use  by  operational  units 
through  such  systems  as  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2),  which  shows  a  map  display  of 
friendly  and  reported  enemy  locations;  Command  Post  of  the  Future  (CPOF),  which  can  show  highly  detailed,  three- 
dimensional  geospatial  intelligence  data;  or  the  network  topology  displays  of  commercial  monitoring  tools  such  as 
WhatsUpGold^^. 

^  The  OTC  simulation  federation  has  demonstrated,  for  instance,  the  ability  to  perform  engagements  between  -test 
players”  in  live  and  constructive  simulations,  enforcing  a  requirement  for  network  connectivity  among  all  the 
players — hence  a  requirement  to  establish  and  maintain  radio  links  with  the  live  players. 

Conversation  between  the  authors  and  the  Future  Combat  System  (FCS)  OTC  Test  Officer,  Mr.  Joseph  Lucidi, 
February  26,  2008.  Mr.  Lucidi  emphasized  the  challenge  of  being  able  to  monitor  data  collection  in  near  real  time, 
such  that  the  tester  can  understand  his  progress  towards  collecting  the  data  required  for  any  given  test.  Hereinafter 
cited  as  lucidi  interview.” 

^  US  Army  Operational  Test  Command  Test  Officers’  Procedures  Manual  (TOPM)  73-301,  Data  Authentication 
Group,  November  2003,  Appendix  B,  Figure  B-1,  page  B-2,  describes  the  Hevels”  of  data  from  a  test  ranging  from 
Level  1  (raw  data)  to  Level  7  (Conclusions),  and  the  procedures  (such  as  -applying  evaluative  military  judgmenf’) 
to  arrive  at  them.  The  tasks  shown  here  are  those  that  the  authors  estimate  are  most  amenable  to  visualization 
support.  Hereinafter  referred  to  as  -TOPM  73-301.” 

"  Ibid. 

^  For  instance,  the  Joint  Network  Node  (JNN)  Initial  Operational  Test  in  2006  produced  over  5  terabytes  (TB)  of 
network  traffic  data. 

“^TOPM  73-301. 

’  DoD  Architecture  Framework,  Version  1.5,  23  April  2007,  Volume  II,  page  4-40  describes  the  OV-5,  its  possible 
formats,  and  the  benefits  and  drawbacks  of  each. 

10  yoPM  73-301  briefly  describes  possible  methods  of  single-  and  multi-factor  analysis — ^but  having  been  written 
before  the  advent  of  extremely  high  data  volumes,  and  before  sophisticated  visualization  means  were  available,  it  is 
silent  on  any  detailed  methods  for  performing  these  steps  under  current  conditions. 

*'  Lucidi  interview. 

J.  J.  Thomas  and  K.  A.  Cook.  Illuminating  the  Path:  The  Research  and  Development  Agenda  for  Visual  Analytics'. 
IEEE,  2005. 

University  XXI  Data  Visualization  Tool  Information  Briefing,  Texas  A&M  University,  October  22,  2008. 

Edward  Tufte,  in  his  classic  Visual  Explanations  (Graphics  Press,  Cheshire,  CT,  1997)  makes  the  point  (p.  45) 
that  all  information  (-a-11  observations  for  all  variables. . .”)  should  be  available  to  the  analytical  audience.  This  of 
course  becomes  extremely  difficult  as  data  volumes  explode,  and  sources  and  formats  proliferate.  Hence,  the  ability 
to  rapidly  query  all  available  data  with  differing  query  parameters  becomes  indispensible  to  the  analyst  of  today’s 
extremely  large  data  sets. 

Interview  by  the  authors  with  Mr.  Sid  Fincher  and  Mr.  John  Fuller  of  the  OTC  FCS  test  team,  April  8,  2008. 


